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ABSTRACT 

Speaking and finger spelling words are dynamic, 
biological, visual activities that serve language 
communication. However, fingerspelling is a 
manual articulation of orthography, whereas the 
visible aspect of spoken language is a product of 
speech articulation. These two stimulus types 
provide a revealing contrast for examining the 
cortical substrate for language perception. 
Lunctional magnetic resonance imaging (fMRI) was 
used to investigate cortical activity due to viewing 
spoken (lipread) vs. finger spelled words. In 
Experiment 1, young adults with prelingual-onset 
severe-to-profound hearing impairments were 
imaged. In Experiment 2, young adults with normal 
hearing and minimal previous experience with 
fingerspelling were imaged. In both participant 
groups, fingerspelling and lipreading activated 
regions of the superior temporal sulcus (STS). 
However, in the normal-hearing participants, 
fingerspelling activated fewer regions in the STS. 
In both participant groups, other activation was 
observed for fingerspelling, including several 
dorsolateral parietal areas. These results suggest that 
the perception of different forms of biological 
motion (spoken and fingerspelled) occurs in 
partially shared but partially distinct cortical 
networks, depending on linguistic significance/ 
experience of the perceiver. 

1. INTRODUCTION 

Studies comparing oral and manual languages 
provide evidence regarding effects of perceptual and 
linguistic experience on the cortical organization for 
perception of language. Studies investigating the 
cortical processing sites responsible for 
understanding manual language have primarily 
focused on examining native users of American 
Sign Language and have provided evidence that 
similar cortical sites are active in deaf, native 
signers viewing ASL as in normal hearing, native 
English users reading and listening to English 
[1,2,3]. These studies of native signers, along with 
the studies of visible and auditory spoken language 
[4,5, 6,7,8] suggest that, at a relatively coarse level 
of analysis, the cortical organization of the language 
system is remarkably similar, despite substantial 



variation in the perceptual and linguistic experience 
of the perceiver. 

In the current study, we focused on the cortical sites 
active during the visual recognition of spoken and 
manually encoded English words. We compared and 
contrasted the cortical activation resulting from 
viewing spoken and fingerspelled words. 
Eingerspelling is a manual encoding of orthography 
used by deaf individuals to communicate words that 
do not correspond to signs. Eingerspelling and 
lipreading provide a particularly interesting contrast 
for examining cortical processing of language: 
Eingerspelling and speech both convey American 
English and share the properties of being produced 
for communication via biological motion. However, 
fingerspelling represents a socially agreed-upon 
artificial code, whereas speech is an aspect of the 
communication system for which humans are 
biologically predisposed. Because individuals differ 
dramatically in their experience with reception of 
fingerspelling, this contrast also affords the 
opportunity to examine effects of lifelong perceptual 
experience on cortical activity. 

Previous studies have demonstrated that 
speechreading activates cortical regions that include 
the superior temporal sulcus and the lateral surface 
of the superior temporal gyrus [4,5]. The cortical 
regions active when viewing fingerspelling have 
not, to our knowledge, been examined. We 
hypothesized that, given their shared stimulus 
properties and linguistic function, a similar pattern 
of cortical activity would be observed for 
speechreading and viewing fingerspelling. 
Experiment 1 investigated whether the perception of 
fingerspelled words results in a similar pattern of 
cortical activity as lipreading in deaf adults who are 
skilled at the reception of both stimulus types. 
Experiment 2 investigated whether the pattern of 
cortical activity for perception of fingerspelled word 
varied as a function of experience with 
fingerspelling. 

2. EXPERIMENT! 

Experiment 1 was designed to 1) provide an initial 
description of the pattern of cortical activity while 
viewing fingerspelled words and 2) Compare this 
pattern of activity with the pattern of cortical 
activation while speechreading words. 
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2.1 Methods 



2. 1.1. Participants 



Participant 

(gender) 


Age 


Etiology 


Pure 

tone 

avg. 

L- 

R 


Age loss 
discovered 


Lipreading 

Screening 

(words 

correct) 


IHl(F) 


25:10 


Ototoxic 

Drug 


98- 

102 


18 mos. 


59.5% 


1H2(F) 


22:4 


Unknown 


103 

-98 


3 mos. 


62.2% 


1H3(F) 


22:7 


Unknown 


101 

-97 


13 mos. 


35.4% 


1H4(F) 


24:8 


Chicken 
pox and 
high fever 


87- 

100 


3 years 


73.2% 


1H5(M) 


25:6 


Spinal 

meningitis 


120 

-120 


24 mos. 


43.9% 



Table 1: Detailed participant information 



The five deaf participants (see Table 1) were all 
right-handed and had no self-reported learning 
disabilities, vision 20/30 or better and self-reported 
use of English as native language. They had bilateral 
early-onset sensorineural hearing impairment greater 
than 85 dB HL pure tone average across 500, 1000, 
and 2000 Hz. They were currently enrolled at 



California State University at Northridge and had 
been educated in a mainstream and/or oral program 
for years > 8. Participants were screened to be skilled 
lipreaders, and they self-reported knowledge and use 
of fingerspelling. 

2.1.2. Procedure 

At least two experiments were performed per 
participant, a lipreading experiment and a 
fingerspelling experiment. In the lipreading 
experiment, a sequence of silently mouthed 
monosyllabic words (stimulus 'on' condition) was 
contrasted with a sequence of colored patterns 
overlayed on a still frame of the same talker's face 
(control condition). To maintain the participants’ 
vigilence during scanning, they performed a one- 
back monitoring task. This task was chosen because 
it could be performed by either a linguistic or non- 
linguistic strategy and could therefore be performed 
by knowledgeable and naive participants. The 
participants pressed a button when they detected an 
immediate repitition of the lipread words or colored 
patterns. In the fingerspelling task, a sequence of 
fingerspelled monosyllabic words was similarly 
contrasted with a sequence of colored shapes 
overlayed on a still frame of the signer's hand. 'The 
static control was chosen because we were 
interested in only examining cortical activation 
related to viewing the motions produced during the 



Lipreading IH4 

HI 
LO 



Lipreading IH3 





Fingerspeiiing 

Figure 1. Four 10mm thick contiguous coronal slices showing activation 
during lipreading and fingerspelling for two deaf participants. 
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spoken or fingerspelling productions. The gradient 
sound was present equally during all conditions. 

2.1.2. Data Acquisition 

A GE 1.5T Signa Horizon MRI system with echo- 
planar imaging (EPI) was used to conduct the fMRI 
experiments. A quadrature head-coil was used to 
acquire a time- series of 500 images from four 10- 
mm thick contiguous transaxial or coronal sections 
(125 images per section) using an EPI sequence with 
the parameters: TR (repetition time) = 4s; effective 
TE (echo-time) = 45ms; 90 deg flip angle; 64x128 
acquisition matrix; 20x40cm" field-of-view; and 
NEX (no. of excitations) = 1. Spin-echo anatomical 
images of the same sections were also acquired 
using TR = 400ms and TE = 14ms to obtain good 
gray/white-matter demarcation. 



method descrihed in detail in [9]. Activated pixels 
were identified based on a statistical parameter 
mapping (SPM) approach, using the software 
package SPM99. A box-car reference function 
matching the task and control presentation sequence 
and delayed by one image (i.e., 4 sec) to account for 
hemodynamic delay was employed. The z-score of 
each pixel (defined as the average intensity 
difference between the task and control condition 
for a given pixel, divided by the standard deviation 
of its intensities during the control condition) was 
computed and pixels with z-scores above a threshold 
(p < .001) were identified. Eor each participant, 
activated pixels were color-coded according to their 
individual z-score and superimposed on 
corresponding anatomy. 

2.2 Results and Discussion 



2.1.3. Image Analysis 

The first five images per slice were ignored to 
establish equilibrium and starting at image 6, the 
task (e.g., lipreading or fingerspelling) and the 
corresponding control condition were presented in 
an alternating sequence for a total of 4 cycles. The 
time-series of images were co-registered to a 
reference image within the time-series using the 



Images clearly showed common regions of cortical 
activity while viewing spoken and fingerspelled 
words including Brodmann areas 21, 22, 37, & 39 
(see yellow boxes in Eigure 1). In four participants 
(IH2, IH3, IH4, & IH5), common activity was 
located in the STS. Eor IHl, common activity was 
located in the middle temporal gyrus. These regions 
activated by fingerspelling are consistent with 
regions observed in studies of both spoken and 



Lipreading NH1 




Fingerspeiiing 



Lipreading nH2 




Fingerspeiiing 



Figure 2. Eour 10mm thick contiguous coronal slices showing activation 
during lipreading and fingerspeiiing for two hearing participants. 
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signed language[l,2,3,4,5,6,7,8]. 

In three participants (IHl, IH2, & IH4), several 
dorsolateral parietal area were strongly activated by 
viewing fingerspelled words (BA 7; see green boxes 
in Figure 1). These areas were not activated by 
lipreading. These areas are likely involved in the 
attention, perception, and memory for a sequence of 
hand motions [10, 11, 12, 13]. 

Taken together, these results suggest that the 
perception of different forms of biological motion 
(spoken and fingerspelled) occurs in partially shared 
but partially distinct cortical networks. 

3. EXPERIMENT 2 

The majority of common activity observed in 
Experiment 1 was located in the STS. Regions of the 
STS have been demonstrated to be responsive to the 
perception of both biological motion and native 
spoken language [4,5,6,7,8,14,15]. At issue in 
Experiment 2 was whether linguistic and/or 
biological motion processing were responsible for 
the common regions of STS activity. One method 
for investigating is to vary the linguistic experience 
of the participants with the reception of 
fingerspelling. In Experiment 2, we used fMRI to 
investigate whether viewing fingerspelled words 
activates the same regions of STS in hearing 
lipreaders who had little or no previous experience 
perceiving fingerspelled words. 

3.1 Methods 

3.1.1. Participants 

Seven individuals between 18 and 40 years of age; 6 
right-handed and 1 individual with mixed 
handedness; no self-reported learning disabilities; no 
self-reported hearing loss, vision 20/30 or better; 
self-reported use of English as native language; 
above-average screened lipreading performance. 
Self-report of minimal or no previous experience in 
the reception of fingerspelling. One participant had 
learned about fingerspelling in grade school, but 
reported not understanding any of the words during 
the imaging session. One participant was not 
included due to minimal activity in both conditions 
at the p < .001 threshold. 

3.1.1. Methods 

All experimental procedures and imaging analyses 
were the same as in Experiment 1 . 

3.2 Results and Discussion 

Images showed activation along the STS while 
viewing spoken words and fingerspelled words (see 
yellow boxes in Eigure 2). However, in contrast to 



the results obtained in Experiment 1, the common 
activity was predominantly located in the most 
posterior regions of the STS. Taken together with 
the results of Experiment 1, the evidence suggests 
that the most posterior regions of the STS are 
responsive to biological motion, and the more 
anterior regions are responsive to linguistic 
attributes of the stimuli. This pattern of results is 
consistent with the literature on cortical activity for 
spoken language and for biological motion 
processing [8,14,15]. 

In all six participants, several dorsolateral parietal 
areas were strongly activated by viewing 
fingerspelled words (see green boxes in Eigure 2). 
Viewing fingerspelled words also resulted in cortical 
activity in several parietal areas. These areas are 
likely involved in the attention, perception, and 
memory for a sequence of hand 

motions [10,11,12,13]. 

4. CONCLUSIONS 

The results of these two experiments suggest that the 
perception of different forms of biological motion 
(spoken and fingerspelled) occurs in partially shared 
but partially distinct cortical networks, depending on 
linguistic significance/experience of the perceiver. 
Greater overlap in cortical activity was observed 
when both forms of biological motion were 
linguistically significant to the perceiver. This result 
suggests that previous observations of STS activity 
for lipreading were due to both biological motion 
processing and linguistic processing. 

Based on results in the literature[10,l 1,12,13], we 
hypothesize that the presence of parietal activation 
could be due to a participant’s processing strategy. 
Eor example, participants demonstrating parietal 
activity are likely attending to and remembering the 
sequence of hand motions and not using a linguistic 
coding of the stimulus. Alternatively, the 
recruitment of parietal regions may be related to task 
difficulty. 
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